Introduction and set up [Bath workshop]¶

Authors:	Andrey V. Brukhno (andrey.brukhno@stfc.ac.uk), John Purton (john.purton@stfc.ac.uk), James Grant (r.j.grant@bath.ac.uk), and Tom L. Underwood (t.l.underwood@bath.ac.uk)

Introduction¶

Welcome to the DL_MONTE workshop in Bath! Here we provide some information which is required before the DL_MONTE tutorials can be undertaken in earnest, namely:

Essential information about DL_MONTE
Information about Balena, the HPC facility you will use for the workshop
How to log on to balena and set up your shell environment in preparation for undertaking the DL_MONTE tutorials

Note that we assume in these tutorials that you are competent in using the unix shell; the unix shell will be the environment in which we will be performing DL_MONTE simulations and analysing their output. If you are not competent at using the shell, then there are a number of excellent tutorials online which can be used to sharpen your skills.

About DL_MONTE¶

DL_MONTE is a state of the art parallel Monte Carlo simulation package aiming to provide the academic community with a general purpose advanced MC engine. The development of DL_MONTE-2 has been instigated by the EPSRC’s Software for the Future programme (research grant EP/M011291/1).

The DL_MONTE GitLab page contains extensive information about DL_MONTE, including links to training material such as these tutorials and a suite of examples. Moreover, the DL_MONTE manual is also an essential resource. A manual is always provided with a DL_MONTE release (see below). Alternatively, a link to the latest version of the manual can be found here.

Balena¶

Balena is the University of Bath’s HPC and all tutorials for this workshop will be run on it. We have reserved some compute nodes on Balena for the workshop, which particpants will be able to use to perform their simulations. Here we provide general instructions for running DLMONTE and associated programs for the visualistation and anlalysis of data.

Logging on to Balena¶

You will have access to the computers in Chancellors’ Building Room 5.13 for the workshop. These are Windows machines. The procedure to access Balena from these machines is as follows:

Ensure the program ‘Xming’ is running by typing ‘Xming’ into the search bar at the bottom left of the Windows desktop, and clicking on the ‘Xming’ logo. Note that after clicking on it nothing will seem to happen. This is normal.
Initiate the program ‘KiTTY’ by typing ‘KiTTY’ into the search bar, and clicking on the ‘KiTTY’ icon.
Click on ‘balena.bath.ac.uk’, and then ‘Start’. This will spawn a terminal.
Enter your username, then your password.

You should now be on Balena.

Setting up the environment on Balena¶

Instructions for users external to the University of Bath¶

Once you have logged into Balena you will need to load the environment to your profile by running the following commands:

[username@balena-01 ~] module load group dl-monte-ws-2020
[username@balena-01 ~] module load workshop/profile_setup

If you log out of Balena (by closing the terminal window or typing the command ‘exit’) and then log back in again your profile will now be set up ready for the workshop.

Note that you only need to do this once; you only need to do this the first time you log in.

Instructions for members of the University of Bath¶

If you are based in the University of Bath, and hence have a Bath computing account, then following the above instructions for external users will alter your default environment so that it corresponds to the workshop. Specifically, it changes your .bash_profile file, which contains commands for configuring your environment. The above instructions will work for Bath members. However, you may not want such a ‘permanent’ change to your environment. With this in mind, running the following commands will undo the changes to your .bash_profile, restoring everything to how it was before the workshop:

[username@balena-01 ~] module load workshop/profile_cleanup

Be sure to run this command at the completion of the workshop to restore your .bash_profile to what it was originally. Note that you must log off and on again before your original .bash_profile is restored.

Alternatively, if you do not want to permanently alter your .bash_profile, then instead of following the above instructions you could run the following commands each time you log in to Balena for the workshop:

[username@balena-01 ~] module purge &> /dev/null
[username@balena-01 ~] module load group dl-monte-ws-2020 workshop/env

This will temporarily configure your environment to be suitable to the workshop; when you log out of Balena the changes to your environment are lost.

Setting up the directories and obtaining tutorial files¶

In your home directory there is a directory named scratch. All simulations should be performed here. The tutorial files are located in a directory which is stored in the shell variable INPUTS. You should copy the contents of this directory to a new directory in scratch using the following commands (or similar):

[username@balena-01 ~] cd scratch
[username@balena-01 scratch] mkdir inputs
[username@balena-01 tutorials] cd inputs
[username@balena-01 inputs] cp -r $INPUTS/* .

Note that in the above commands inputs was the name chosen for the directory which houses the tutorial input files. Note also the necessary ‘$’ in front of INPUTS.

Looking at the contents of the inputs directory, it can be seen that there are subdirectories corresponding to each tutorial:

[username@balena-01 inputs] ls
tutorial_1  tutorial_2  tutorial_3  tutorial_4  tutorial_5  tutorial_6  tutorial_7  tutorial_8  tutorial_9

Running DL_MONTE¶

You are almost ready to work through the first tutorial. All that remains is how to actually run the simulations.

DL_MONTE has three input files: CONTROL, CONFIG and FIELD. There are a few mechanisms for running DL_MONTE during this workshop. In all cases you should first be in the directory containing these input files.

Job scripts¶

A job script can be used to submit a simulation job to Balena. We have prepared job scripts for performing DL_MONTE simulations during the workshop. The scripts are located in a directory stored in the environmental variable SCRIPTS:

[username@balena-01 ~]$ ls $SCRIPTS
disp.sh  parallel.sub  single.sub       strip_gcmc.sh  time.sh                    unbiasedhist.sh
hist.sh  probratio.sh  strip_adsorb.sh  strip_yaml.sh  unbiasedhist_probratio.sh  volmoves.sh

The relevant files are single.sub and parallel.sub, corresponding to a serial DL_MONTE simulation and a parallel DL_MONTE simulation (using 4 cores), respectively.

To launch a serial job we must copy the single.sub script into the directory containing the DL_MONTE input files, and then use the sbatch command as demonstrated below:

[username@balena-01 tutorialX]$ ls
CONFIG  CONTROL  FIELD
[username@balena-01 tutorialX]$ cp $SCRIPTS/single.sub .
[username@balena-01 tutorialX]$ ls
CONFIG  CONTROL  FIELD  single.sub
[username@balena-01 tutorialX]$ sbatch single.sub
Submitted batch job 2471728

The job number, 2471728 here, will be different for your job. The sbatch command causes the simulation job, defined in the jobscript, to be put in a queue on Balena, and the simulation is run remotely when there are enough computing resources available. The status of the the job can be checked using the following command:

[user@balena-01 tutorialX]$ squeue -u $USER
     JOBID       NAME     USER    ACCOUNT    PARTITION    ST NODES  CPUS  MIN_MEMORY   GRES          START_TIME  TIME_LEFT PRIORITY NODELIST(REASON)
   2471728 single.sub     user prj-dlm002        batch     R     1    16         62G (null) 2020-01-26T15:36:34    9:59:45   777689 node-sw-087

If for whatever reason you need to cancel the job use the scancel command followed by the job ID:

[user@balena-01 tutorialX]$ scancel 2471728

The early tutorials should run in a few seconds, and if the simulation exits successfully the command:

[user@balena-01 tutorialX]$ ls -l

should reveal the output files created by the simulation:

OUTPUT.000
ARCHIVE.000
PTFILE.000
RECVON.000
REVIVE.000
RSTART.000
YAMLDATA.000

Parallel job¶

A parallel (MPI) DL_MONTE simulation (using 4 cores) can be run by using the parallel.sub script instead of the single.sub script. However, this is not usually necessary in the tutorials - unless told otherwise please use the single.sub submission script described above:

Interactive session¶

Finally, as opposed to submitting jobs to the queue, DL_MONTE could be run interactively. The command

[user@balena-01 tutorialX]$ sinteractive --time=6:00:00

spawns an interactive session (for 6 hours). This means that you are logged on to one compute node of Balena, and are free to run commands as if it were ‘your own machine’ (for 6 hours, at which point the session will expire). Within an interactive session to start a serial DL_MONTE simulation one should call the DL_MONTE executable DLMONTE-SRL.X in the directory containing the input files:

[username@intnode tutorialX]$ ls
CONFIG  CONTROL  FIELD
[username@intnode tutorialX]$ DLMONTE-SRL.X &

Note that above we have put & at the end of the call to DL_MONTE to run the simulation in the background. If we did not then we would not be able to run any more commands until the DL_MONTE simulation had completed.